Show&Tell: A Semi-Automated Image Annotation System

A multimedia system for semi-automated image annotation, Show&Tell combines advances in speech recognition, natural language processing, and image understanding. Show&Tell differs from map annotation systems and has tremendous implications for situations where visual data must be coreferenced with text descriptions, such as medical image annotation and consumer photo annotation. S how&Tell takes advantage of advances in speech technology and natural lan-guage/image understanding research to make the preparation of image-related information more efficient. Specifically, we aim to identify relevant objects and regions in the image, as well as to attach text descriptions to them. We use a combination of automated and semi-automated image understanding tools in object and region identification. Image analysts can use Show&Tell in applications where text descriptions must be corefer-enced with image areas, such as medical image annotation, National Aeronautics and Space Administration (NASA) space photo annotation, and even consumer photo annotation. Medical images suit our system well, since radiologists already employ speech to dictate their findings and robust image understanding technology is available for several areas, such as chest and lung radiographs. In a joint effort with Kodak, we are adapting our system for consumer photo annotation. Since still cameras can be fitted with microphones, speech annotation of photos is now possible. Consumers will be able to easily create searchable digital photo libraries of their pictures and focus primarily on pictures of people in various contexts. Multimedia input analysis Multimedia systems involving speech and deic-tic input can be classified into two major categories: multimedia input analysis and multimedia presentation. Our work focuses on the former. Our system differs from previous work in the area of adding text annotations to pictorial data in the following ways. Most systems assume that there already exists an underlying semantic representation of the pictorial data. We don't. We used Clark's terminology. 1 The region to which the user points is the demonstratum, the descriptive part of the accompanying text is the descrip-tor, and the region to which the user intends to refer is the referent. Much of the recent work in multimedia input analysis concerns disambiguat-ing ambiguous deictic references, that is, determining which of the possible referents that map to the same demonstratum is intended by the user. 2 Accompanying linguistic input, in the form of speech, is used for this purpose. Such systems assume that the type of deixis being used, known as demonstratio ad oculus, is distinguished by the fact …